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Executive Summary 



This study is an examination of student use of the PLATO system in Highland Park 
Independent School District (Highland Park ISD). This district is located in the center of 
Amarillo, Texas, and includes one elementary school, one middle school, and one high 
school. The total student population numbers approximately 860. The following 
description of the PLATO program is based upon phone interviews with the Assistant 
Superintendent who serves as Site Coordinator, three written surveys from district 
teachers, and three surveys from district students. The data on PLATO use come from 
system files; standardized test scores come from the Texas student assessment system. 

Program Description. PLATO was brought to Highland Park District by their new 
Superintendent; its use is monitored by the Assistant Superintendent/Site Coordinator. 

The program has been used by high school students for four years and middle and 
elementary students for two years. Student scores on the Texas achievement test are 
monitored by the district, as are the teacher PLATO reports that are required every six 
weeks. Principals oversee PLATO use at the building level. The district has provided 
teacher training and PLATO support. 

All students in grades 3 through 12, including special education students, use PLATO. 
The district standard for student PLATO use is two hours per week, evenly divided 
between math, language arts, and science and social studies. Teachers are expected to 
assign students to PLATO modules based upon their performance on state and local tests 
and learning objectives. Instructors are also expected to correlate PLATO assignments 
with classroom instruction and to monitor student progress. The building administrators 
monitor PLATO use in their schools. 

Data Analysis. Student use of PLATO is presented in this report as time spent on the 
system and the number of modules mastered. As currently available, student data is 
aggregated for all three schools. 

For the 1999 and 2000 school years students mastered an average of 48 PLATO modules 
per year; they spent an average of 18 hours engaged in PLATO learning per year. The 
level of PLATO use was significantly higher in 1999 than 2000: 50 modules mastered in 
1999 compared to 28 in 2000; 20 14 engaged hours in 1999 compared to 7 14 hours in 
2000. 

The Texas Assessment of Academic Skills (TAAS) is a criterion-referenced measure of 
student achievement in the statewide curriculum for mathematics, reading, and writing. 
The TAAS scores are reported using the Texas Learning Index (TLI). The TLI provides a 
standardized achievement score which facilitates comparisons of scores across grades or 
over time. The TLI were used to report student achievement in math, reading, and writing 
in Highland Park. 
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The Highland Park math and reading scores were somewhat higher than the statewide 
averages on these tests; the district’s writing scores were somewhat lower than state 
averages. 

The analysis of PLATO use data found that increases in the number of PLATO modules 
mastered are positively related to higher test scores. The more PLATO modules students 
mastered, the higher were their TLI scores. 

The study also found that taking more hours on PLATO without making progress in 
mastering modules is negatively related to higher test scores. This suggests that slow 
progress in mastering modules on PLATO can be used as an indicator of learning 
problems that teachers should note and address. 




Highland Park, TX 



Page 3 



Table of Contents 



Executive Summary 2 

Introduction 5 

Program Description 5 

The District 5 

Technology in the District 6 

Support of PLATO 6 

Teacher Preparation 6 

Program Design 7 

Student Placement in PLATO 7 

Student Computer Use 7 

Monitoring Student Achievement 8 

Learning Materials 9 

Teacher PLATO Improvements 9 

PLATO Improvements 9 

Teacher and Student Agreement 9 

Data Analysis 11 

PLATO Use and Test Score Findings 1 1 

Sample 11 

PLATO Use Data 1 1 

Texas Learning Index (TLI) 12 

Differences between 1999 and 2000 Use and Test Data 15 

Relating PLATO Use to Test Scores 17 

Instructor’s PLATO Evaluation 21 

PLATO Content 21 

Teacher Experience with PLATO 24 

Student Experience with PLATO 25 

Frequency of Activities 26 

Conclusions & Discussion 27 

About the Authors 29 



Highland Park, TX 



Page 4 



Highland Park independent School District 



Introduction 



Highland Park ISD has adopted the PLATO system as part of a strategy to improve 
individual student learning and to raise their scores on the Texas state assessment tests in 
mathematics, reading, and writing. This is a study of PLATO as part of the regular school 
instruction in upper elementary, middle, and high school classes. Special attention is 
given to the use and impact of PLATO on mathematics, reading, and writing performance 
during the 1998-1999 (1999) and 1999-2000 (2000) school years. 

The following report first describes Highland Park — the process they used in 
implementing PLATO, the support they have provided for PLATO, and teacher 
preparation for technology. This report describes the use of PLATO in school in terms of 
program design, student placement in PLATO, student computer use, and the monitoring 
of student computer use. The program descriptions were taken from these sources: 

Phone Interview with Program Coordinator — Site Profile Interview Questions (1) 
Written responses — PLATO Instructor Interview Questions (3) 

Written responses — Instructor’s PLATO Evaluation (3) 

Written responses — PLATO Learner’s Survey (3) 

Secondly in this report, the “Data” section describes the various evaluation activities and 
their results from the following sources: 

Student Use Data and Outcome Findings (806) 

Instructor’s PLATO Surveys and Interviews (3) 

Thirdly, the Conclusions section discusses the findings and relates them whenever 
possible to the overall picture of this PLATO application. 



Program Description 

The District. Highland Park district is located in the panhandle area of Texas where 
the southern plains meet the western desert. Highland Park ISD is a relatively small 
school district, serving a community in the city of Amarillo, Texas. The U.S. Department 
of Education describes Highland Park as being located within a mid-size city. 

The district has one high school, middle school, and elementary school. It employs 65 
teachers and enrolls 862 students. The average teacher-student ratio is 1 to 13. For the 
2000 school year, high school enrollment was 204, middle school enrollment was 215, 
and elementary enrollment was 443 students. The majority of students are white (72%); 
Hispanic students comprise 23% of the student body. Three percent or less of the school 
populations is Black, Asian, or American Indian/Alaskan. As an indicator of community 
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economic status, 51% of elementary students in the district qualify for free or reduced 
price lunch. 

The Assistant Superintendent and PLATO site coordinator, Jamie Carol, was interviewed 
following the Site Profile Interview Questions. Ms. Carol arranged for instructors to 
complete in writing the instructor’s survey, and the instructor’s interview questions as 
well. Her choices were three teachers, one from each school. The teachers, in turn, 
selected one student in each of their classes to complete the Learner’s Survey; two of the 
three students were not designated as to gender, so for this report they are referred to as 
males. The following program information is derived from these interviews and teacher 
and student surveys. 

Technology in the District. Highland Park has been using the PLATO 
program in the high school for the last four years and at the middle and 
elementary 1 levels for two years. At the elementary level, they have been using 
the Jostens programs for the past seven years. The choice of PLATO was a 
district decision based on desires to achieve high educational goals in their 
district. Ms. Carol said, “We believe that differentiated instruction leads to higher 
student learning. Our strategy is to increase the TAAS score by meeting 
individual student needs.” 

Ms. Carol continued: “Our district superintendent had experience with the PLATO 
system in his prior assignments. When he became our superintendent he suggested that 
we would adopt the PLATO system. He also suggested the PLATO curricula to use.” 

Support of PLATO. The technology department in the district takes care of all of 
technical support for the PLATO system. The district has two full-time technology 
specialists to support PLATO. In addition, at the schools ten staff members have been 
assigned to support teachers. The school-level support staff members are each budgeted 
to provide eight days of support per year. 

Teacher Preparation. Generally speaking, district teachers have had four or more 
years of computer experience. However, due to staff turnover, this does not necessarily 
increase over time. No specific information was provided about teacher PLATO training. 
However, this sample of three said that they felt well trained; only one was interested in 
more. 

In this study, the elementary school respondent teaches fifth grade math, science and 
social studies; she has used computers in education for 12 years and PLATO for four 
years. The middle school interviewee teaches reading and language to 6 th and 8 th grade 
students; she has used computers for four years and PLATO for just one year. The high 
school respondent teaches social studies and uses PLATO for U.S. History; he has used 
computers for four years, and PLATO for three. 



1 The PLATO products discussed in this report do not include the elementary-level courseware recently 
added through acquisition of Wasatch Interactive, Inc. — ed. 
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Program Design. Grades three through twelve in Highland Park use PLATO 
throughout the school year. All students in the grades three through 12 use the PLATO 
system — both general education and special education students. Across the district, there 
are approximately 600 PLATO users per year. 

The district requires that students spend two hours a week on PLATO or cover 6 to 9 
modules in this time period. This time should be distributed approximately as: 

• 45 minutes of math 

• 45 minutes of language arts 

• 45 minutes of science and social studies 

There is no maximum time limit on using PLATO other than the constraints of student 
time and system availability. 

The Assistant Superintendent oversees the district use of Plato. She says, “We have staff 
turn in reports every six weeks on Plato use to encourage their use of the system and keep 
students moving through the modules.” The regular school administrators supervise the 
teacher use of Plato on an ongoing basis in each school. 

Student Placement In PLATO. The district creates its own learning paths using 
the PLATO Pathways management system. District and school personnel look at the state 
TLI student test scores every year to determine areas of skill or informational strengths 
and deficiencies. At the beginning of each semester, in September and January, students 
are assessed for their knowledge on TAAS objectives. In addition, student PLATO 
computer reports are reviewed every six weeks. Based upon this information, students are 
assigned by their teachers to modules that address their learning needs, particularly those 
that are aligned with the TAAS and demonstrated on PLATO reports. Instructors are 
responsible for assigning students appropriately and for synchronizing PLATO 
instructional activities with other educational activities. In order to do these things, 
teachers need to know how to use reports and test data and how to use the PLATO 
management system. 

Student Computer Use. The Assistant Superintendent feels that the students 
generally know the mechanics of a how to use the PLATO system; however, some 
teacher responses indicate that some students do have some difficulty. 

Students use PLATO as part of their regular classroom learning experience, with their 
regular classroom teachers overseeing their PLATO work. Most students work alone, 
one-on-one with the system. Ms. Carol says that some of their geometry classes are 
starting to use the system in whole group instruction, to pre-teach students on concepts in 
the PLATO curriculum and to clarify points that students have had trouble with in the 
past. The high school social studies teacher said that he sometimes uses group work. 

Learners who fail a mastery test are required to study the corresponding tutorial before 
re-taking the mastery test. No other comments were written about monitoring student 
completion of lesson parts. 
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Learners are allowed to “place out of’ a module by taking the mastery test first, but they 
cannot test out of the TAAS objectives. Testing out is a fairly common practice in 
language arts and math; it is not so common in science and social studies. Ms. Carol 
shared the following: “It is really an ongoing effort to make sure that the staff 
understands that it is okay to accelerate students through the curriculum. However, our 
staff needs more training to implement the strategy.” This sentiment was echoed by one 
of the teachers. 

According to one teacher interview form and the corresponding student survey, it seems 
that students in that class are held to the same learning pathways, and students cannot 
proceed, even though they have mastered the material. This situation has created 
dissatisfaction for the student, since he feels that he knows the material and has gained 
little from his PLATO work, at least in math, writing, and science. From the teacher’s 
point of view, she tries to match the PLATO objectives with the areas in which students 
have had low test scores. Her survey indicates that she feels she is able to make 
appropriate assignments, although the twice-changed answer indicates that she needed to 
think about it. In a later question, she confided: “PLATO serves a great purpose for 
reinforcement, practice, and tutorials for older students. It’s difficult to find the proper 
level of study for intermediate age kids in an elementary school (3-5). ” 2 

None of the three teachers who were interviewed said that they used PLATO for an 
introduction to a new lesson or unit or for placement or assessment of students. The high 
school teacher uses the program during the activity or information phase of a lesson. All 
three respondents say that they use it for practice, review, reinforcement or transfer 
(application) phase. In another response, however, one teacher wrote that the classroom 
rotation actually made PLATO time more of an assessment. The elementary teacher, in 
her three subject areas, specified that she used it for reinforcement. The middle and high 
school teachers reported that PLATO was available any time as a resource, but they did 
not include details. 

Monitoring Student Achievement. The district monitors student progress on 
PLATO by requiring reports every six weeks. On the survey, all three teachers wrote that 
they use PLATO reports to monitor student work. In addition, the elementary teacher 
monitors students by observing them on the computer somewhat and using time-on-task 
reports, however, most of her time is spent with other students during “Center time” or 
reading the Weekly Reader to them. The high school teacher monitors students while 
they work, answering questions and problems and asking what they have learned. The 
middle school teacher usually works with other students during computer time. 

In order to encourage consistent student work, one teacher observed student computer 
work and two gave grades for PLATO work. The middle school teacher gave grades for 
the scores and number of modules completed. The high school teacher gave the PLATO 
average the weight of a major test grade. 



2 The recently-added PLATO Primary curricula, not discussed in this report, are age-appropriate for 3-8. - 
ed. 
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Learning Materials. All of the teachers who were surveyed use textbooks and Web 
sites along with PLATO. None of them uses PLATO paper products. 

Teacher PLATO Improvements. One teacher suggested on the open-ended item 
that she could improve her use of PLATO by assigning it in smaller units so that she 
could display student work more clearly. Indeed, reports on smaller units could help her 
to adjust her classroom instruction as well as monitor student success. She also wrote that 
she should tie PLATO more to the lesson in class. The middle school teacher said that 
perhaps she could give more description of the PLATO lesson. 

PLATO Improvements. Overall, these teachers like most the reinforcement and 
review of concepts they have taught in their classrooms. They had separate major dislikes 
of the program. The elementary teacher said that students did not know how to answer 
questions and that they did not like working on the program. 3 The middle school teacher 
said that lack of time was her biggest problem. The high school teacher feels that there is 
not enough information on government and economics for his needs. In addition, the 
teachers made the following suggestions: 

Elementary Teacher— Make them more age-appropriate, with more animation and color 
for elementary students. Make directions easier to understand [for students], . .It is 
difficult to find the proper level of study for intermediate age kids in an elementary 
school (3-5). 

Middle School Teacher — More consistency in the scoring system. 

High School Teacher — More modules for government and economics. Make the world 
view section part of reports. [PLATO] seems to work for most students. 

Teacher and Student Agreement. In one instance of elementary school, the 
teacher and student agree that the PLATO instruction is not closely aligned with 
classroom instruction. The teacher reports that students do not generally understand the 
explanations, but that the program does involve students. A student from her class 
provides a contrasting point of view by saying he knows the information and feels there 
is not enough involvement. He reports trying hard to leam from the computer, but he 
thinks the lessons are not interesting and would not recommend the program to others. He 
does not want to do more [at least under the current situation.] His problem is definitely 
not the result of problems with the computer in general; he says he feels comfortable 



3 The PLATO courseware reviewed here includes content taught at grade levels 3-14. However, this 
courseware is designed for adults and young adults, and use below upper elementary or middle school 
generally is not recommended by PLATO Learning, Inc.. The PLATO Primary courseware, which has 
been recently added, is age-appropriate for lower elementary. However, it is not a subject of this report. — 
ed. 
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using computers and that they do not affect his self-esteem. The teacher believes that 
students do not receive PLATO well, and this student confirms her opinion . 4 

The middle school instructor was upbeat in her responses and positive about the program 
and the student experience with it. She feels pressed for time, and says she should give 
more introductions to PLATO modules. However, with the rotations on PLATO, it seems 
more like an assessment than a true learning experience for students. The student 
surveyed in her class reports feeling comfortable in using the computer. She is likewise 
positive in his responses and says she tries hard and that the experience makes her feel 
confident. Fun in using the computer appears to be her primary motivation and wrote that 
sometimes the lessons are “boring”, especially in science. She has learned a lot from 
PLATO in all subjects but social studies, about which she was moderately positive in her 
opinion. 

The high school U. S. History teacher is positive about PLATO, although he would like 
more depth in the modules and assessments and more social studies content. He is 
“neutral” about whether students respond well to PLATO (which could indicate a mixture 
of opinions, depending upon lesson content and students’ abilities/interests/personalities). 
His student says he can get into the program and understands what it teaches, but that he 
cannot stop and start when he wants to; neither can he get help when he needs it. He feels 
that he is learning what he needs to but that PLATO does not necessarily match 
classroom instruction. He reports that he does not try hard and, indeed, many of his 
responses are “neutral”. He feels most strongly about being able to go back in lessons to 
review and does not like it when he cannot. He would also like fewer questions at the 
ends of lessons, probably indicating that he does not understand the role of assessment in 
learning or that the teacher is not using assessment as an instructional support. He learned 
least from math, some from reading and writing, and more from social studies and 
science. 



4 This reaction is often indicative of inappropriate placement or alignment decisions, so students are 
compelled to work on modules which are not appropriate to their level. Since the secondary-level 
courseware was used here at the primary level, it is an expected finding. — ed. 
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Data Analysis 



PLATO Use and Test Score Findings 

Sample. The Highland Park dataset included student records from the 1999 and 2000 
school years. Records for a total of 806 students were available for the evaluation. Of 
these students, 471 had complete or partial data for both years and 334 had complete or 
partial data for one of the two years. Data from both groups were included in the analysis 
as appropriate. 

PLATO USG Data. The Highland Park computer network collected PLATO use data 
for all participating students. The PLATO system data included: 

• Number of PLATO modules mastered 

• Total hours of PLATO activity engaged time 

This data does not include modules, if any, that students may have studied in years earlier 
than 1998-1999 or 1999-2000 reports. Nor does it assure that modules studied were 
necessarily related to the test scores available. 

In the analysis a student’s record was used only if it had both PLATO use data and test 
scores within a given year. Since some students did not have complete records on use or 
test scores, different subsets of students were used for each type of test: math, reading, 
and writing. While the students in each group were very similar on their level of PLATO 
use, the analysis described PLATO use separately for the subsets of each test. 

The average number of modules mastered per year by the students studied was 48 (math 
47, reading 48, and writing 49). The number of modules mastered ranged from a low of 
zero to a high of 493 modules. The average number of engaged hours using PLATO was 
about 18 hours (math 19 hours, reading 19 hours, and writing 17 hours). Hours engaged 
in PLATO ranged from near zero hours to 155 hours — equivalent to one person month of 
full-time use. Table 1 gives descriptive statistics for PLATO use; Figure 1 shows the 
average use by test subgroups 



Table 1. Student PLATO Use by Test Groups for Both Years 



Variables 


Modules Mastered With 


Engaged flours With 


Math 

Scores 


Reading 

Scores 


Writing 

Scores 


Math 

Scores 


Reading 

Scores 


Writing 

Scores 


Number of 


Valid 


725 


726 


541 


725 


726 


541 


Cases 


Missing 


81 


80 


265 


81 


80 


265 


Mean 




47.4 


47.5 


48.5 


18.5 


18.7 


17.1 


Std. Deviation 




58.7 


59.2 


63.8 


20.8 


21.5 


21.9 


Minimum 




0 


0 


0 


0.02 


0.02 


0.02 


Maximum 




493 


493 


493 


132.2 


137.1 


155.4 




10% 


4 


4 


2 


1.8 


1.8 


1.5 




25% 


12 


12 


8 


4.8 


4.9 


3.7 


Percentiles 


50% 


27 


27 


26 


10.8 


10.8 


8.9 




75% 


61 


61 


65 


24.5 


24.5 


21.7 




90% 


111.2 


110.4 


122 


47.0 


47.0 


42.2 
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Figure 1. Average PLATO Use Measures for 
Subgroups by Test Scores 




Modules Mastered Engaged Hours 



Texas Assessment of Academic Skills (TAAS). The Texas Assessment of 
Academic Skills (TAAS) is the state’s criterion-referenced test for measuring learning 
outcomes; it is part of the accountability system. TAAS measures the statewide 
curriculum in reading and mathematics at grades 3 through 8 and the exit level; in writing 
at grades 4, 8, and the exit level; and in science and social studies at grade 8. Spanish- 
version TAAS tests are administered at grades 3 through 6. Satisfactory performance on 
the TAAS exit level tests is prerequisite to a high school diploma. In Highland Park, 
students were given the TAAS for math and reading every year from 3 rd through 8 th grade 
and 10 th grade. Writing TLI scores are available for grades 4, 8, and 10. 

Texas Learning Index (TLI). Developed for use with the TAAS, the Texas 
Learning Index (TLI) is a way of interpreting math and reading test results. This state- 
developed statistical index (a scale score) makes it easy to compare student TAAS scores. 
The student outcome measures used in this evaluation were taken from the TAAS results 
and TLI comparisons in math, reading, and writing. A TLI score of 70 on the math and 
reading corresponds to the passing standard, the minimum expectation level, at each 
grade level. A writing TLI of 1,500 is the passing standard. 

Highland Park math scores are somewhat higher than the state’s average year 2000 math 
scores; the median Highland Park score is higher than that of approximately two-thirds 
(62%) of other Texas schools. Highland Park reading scores are about the same as the 
general Texas reading scores. The Highland Park median is above 57% of state reading 
scores. On the other hand, Highland Park writing scores are somewhat lower than the 
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Texas writing scores; Highland Park median writing score was above only 37% of state 
writing scores. 



Table 2. Highland Park Student 1 


rest Scores for Both Years 


Variables 


Test Scores 


Reading TLI 

Math TLI Score Score Writing Score 


Number of Cases _ _,^ a ^ 

Missing 


445 446 221 

361 360 585 


Mean 

Std. Deviation 


83.7 86.8 1667.2 

6.8 9.5 230.4 


Minimum 

Maximum 


53 47 640 

93 101 2490 


10% 

25% 

Percentiles 50% 

75% 

90% 


75 73 1500 

80 83 1550 

86 89 1630 

89 94 1760 

91 97 1836 



The following figure — a “box plot” — demonstrates the range of the majority of test 
scores for both Highland Park and the state of Texas. The numbers in the boxes show the 
scores attained by the middle 50% of the students. The line that extends downward out of 
the boxes (“whiskers”) end at the point below which 10% of the students scored; the lines 
extending upward end at the point below which 90% of students scored. For example, the 
first gold colored box shows the distribution of Highland Park math scores. The middle 
half of the scores were between 80 and 89 points. Ten percent of the students scored 
below 75 points; ten percent also scored above 91 points. The minimum passing level, 70 
points, is marked with a red line. 
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Differences between 1999 and 2000 Use and Test Data. Having evaluation 
data for two school years allows us to ask whether there have been changes in the level of 
PLATO use or in student outcomes from year to year. Interviews with district personnel 
indicated that the level of PLATO use had changed from 1999 to 2000, and that there was 
a question of whether this might have affected student outcomes. 

Whole-group comparisons. A comparison of 1999 data with the 2000 data showed that 
there was a significant difference in PLATO use between the two years. In 1999 there 
were significantly more modules mastered and more hours of students engaged in 
PLATO learning. These differences were relatively large, equaling about a one-half 
standard deviation change. For the 1999 and 2000 school years students mastered an 
average of 48 PLATO modules per year; they spent an average of 18 hours engaged in 
PLATO learning per year. The level of PLATO use was significantly higher in 1999 than 
2000: 50 modules mastered in 1999 compared to 28 in 2000; 20 Vi engaged hours in 1999 
compared to 7 l A hours in 2000. 

On the other hand, a comparison of TLI math, reading, and writing scores for all students 
in 1999 compared to 2000 showed no significant difference in test scores between the 
two years. All differences were within the limits expected for simply random variation in 
scores from year to year. 
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Table 3. Comparison of Use and TLI Scores for 1999 and 2000 School Years 



Variable 


Year 


N 


Mean 


Std. 

Deviation 




df 


Sig. 

(2-tailed) 


Modules 


1999 


674 


39.49 


49.97 


7.93 a 


1082.52 


0.000 


Mastered 


2000 


602 


21.72 


28.13 


Hours 


1999 


674 


15.63 


20.27 


8.91 a 


858.85 


0.000 


Engaged 


2000 


602 


8.20 


7.23 


Math 


1999 


277 


83.20 


7.53 


- 1 . 60 a 


527.07 


0.111 


Score 


2000 


353 


84.09 


6.15 


Reading 


1999 


281 


86.65 


9.64 


-0.99 b 


631 


0.320 


Score 


2000 


352 


87.39 


8.89 








Writing 


1999 


84 


1635.94 


155.82 


-1.78 a 


218.70 


0.076 


Score 


2000 


137 


1686.41 


264.66 





a Equal variances not assumed for /-test 
b Equal variances assumed for /-test 



Matched-group comparisons. A different look can be taken at the issue of year-to-year 
change by looking only at students who have test scores in both 1999 and 2000. This 
matched-group approach is usually more sensitive to small differences than just looking 
at whole-group comparisons. However, the findings do not change very much with this 
analysis approach. There were similarly higher levels of PLATO use in 1999 as 
compared to 2000; if anything, the engaged time difference was even larger (.75 standard 
deviations) than it was for the whole-group comparison. 

On the TLI scores, the matched-group analysis shows a small increase (.23 standard 
deviation) in math scores from 1999 to 2000. This difference may be due to many factors 
such as testing differences or differences in instruction. Reading scores did not change 
significantly between the two years. For any given student, TLI writing scores were only 
available for one of the two years so no matched-group writing comparison was possible. 



Table 4. Matched-Group Comparisons across 1999 - 2000 



Paired Samples 
Statistics 


N 


Std. 

Mean Deviation 


Cor- 

relation 




Modules 


1999 


471 


43.0 


52.5 


0.25 l a 


7.99 a 


Mastered 


2000 


471 


23.2 


30.2 






Engaged 


1999 


471 


17.3 


21.3 


0.350 a 


1 0. 19 a 


Hours 


2000 


471 


7.9 


6.5 






Math 


1999 


186 


83.7 


6.8 


0.628 a 


-3.56 a 


Score 


2000 


186 


85.1 


5.5 






Reading 


1999 


188 


87.7 


8.6 


0.653 a 


-1.23 


Score 


2000 


188 


88.4 


9.1 







Significant at p < .001 
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Relating PLATO Use to Test Scores. The primary analysis of the outcome 
evaluation was the comparison of the level of PLATO use and TLI math, reading, and 
writing scores. The question being examined is how variations in the number of PLATO 
modules mastered or in hours engaged in PLATO relate to test scores. 

A multiple regression approach was used to look at how mastery and time related to test 
scores. After doing a preliminary analysis of scatterplots and residual scores, it was 
decided to look at several independent variables in the regression model: 

M Total number of PLATO modules mastered 
M 2 Squared number of modules mastered 
H Total engaged hours for all PLATO use 
H 2 Squared number of engaged hours 
M * H Interaction of modules and hours (product of M times H) 

The following regression model was then run for each of the three TLI test scores: 

Predicted test score = Constant + M + H + M 2 + H 2 + M*H 

Finally, reduced regression models were computed by keeping only the independent 
variables that made a statistically significant contribution to predicting the test scores. 

The general finding of the regression analysis is that increases in number of PLATO 
modules mastered is positively related to higher test scores. The more PLATO modules 
mastered the higher the TLI scores. 

The second general finding of the analysis is that greater numbers of engaged hours on 
PLATO are negatively related to higher test scores. The more hours spent on PLATO, 
particularly when the number of modules is controlled for, the lower the TLI score. This 
suggests that slow progress in mastering modules on PLATO is an indicator of learning 
problems that the teacher should note and address. 

As with other analyses predicting student test scores with a few programmatic variables, 
this analysis accounted for only a modest portion of the variability in test scores. The 
math regression model predicts 1 1 percent of the variability in TLI math scores. The 
reading regression model predicts 6 percent of the variability in TLI reading scores. The 
writing regression model predicts 21 percent of the variability in TLI writing scores. 

As an aside, the math test, and to lesser degrees the reading and writing tests, showed 
“ceiling” and “floor” effects. Given that the tests were designed to measure adequate 
academic progress around a “passing” score, and not to diagnose the full range of student 
ability, this is not a problem for the state’s use of the test. However, one consequence of 
topping or bottoming out on test scores for the evaluation is that a graph of the 
relationship between test scores and PLATO use measures shows some leveling off at 



Highland Park, TX 



Page 1 7 



high or low levels. This does not necessarily mean that low or high ability students are 
not benefiting from PLATO instruction; it more likely means that the test is not capable 
of measuring the effects of instruction outside of a restricted ability range. 

The following tables present the reduced regression models for math, reading, and 
writing TLI scores in Highland Park. The following figures show the predicted response 
surfaces relating the two PLATO use variables to test outcomes. Predicted test scores are 
determined by identifying the point on the colored surface directly above the point where 
the PLATO use variables intersect. The regression formula is more convenient for 
calculating particular predicted scores; the response surface graph provides an overall 
picture of the relationship between the variables. 



Table 5. Multiple Regression Model for Predicting Math TLI Scores from PLATO 
Use 



Dependent 

Variable 


Model Elements 


Unstandardized 

Coefficients 

Std. 

B Error 


Standardized 

Coefficients 

Beta 


t 


Sig. 


Math TLI 


(Constant) 


83.18 


0.62 




134.4 


0.000 


Score 


PLATO Modules Mastered 


0.06 


0.01 


0.60 


4.7 


0.000 


R 2 = .1 14 


Engaged Hours 


-0.14 


0.03 


-0.48 


-5.2 


0.000 




M 2 


0.00 


0.00 


-0.38 


-3.1 


0.002 




M* H 


0.00 


0.00 


0.31 


2.7 


0.008 




Figure 4. 
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Table 6. Multiple Regression Model for Predicting Reading TLI Scores from 
PLATO Use 



Dependent 

Variable 


Model Elements 


Unstandardized 

Coefficients 

Std. 

B Error 


Standardized 

Coefficients 

Beta 


t 


Sig. 


Reading 


(Constant) 


84.71 


0.78 




108.3 


0.000 


TLI Score 


PLATO Modules Mastered 


0.07 


0.02 


0.49 


3.8 


0.000 


R 2 = .055 


Engaged Hours 


-0.06 


0.02 


-0.15 


-2.6 


0.011 




M 2 


0.00 


0.00 


-0.23 


-2.0 


0.047 




Figure 5. 
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Table 7. Multiple Regression Model for Predicting Writing TLI Scores from 
PLATO Use 



Dependent 

Variable 


Model Elements 


Unstandardized 

Coefficients 


Standardized 

Coefficients 


t 


Sig. 


B 


Std. 

Error 


Beta 


Writing 


(Constant) 


1675.93 


37.36 




44.86 


0.000 


Score 


PLATO Modules Mastered 


3.09 


1.09 


1.06 


2.82 


0.005 


R 2 = .207 


Engaged Hours 


-10.44 


3.15 


-1.22 


-3.31 


0.001 




H 2 


0.12 


0.06 


1.44 


2.02 


0.045 



Writing Predicted Score by PLATO Modules Mastered & Engaged Time 

LLR Smoother 




Figure 6. 
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Instructor’s PLATO Evaluation 



The three teachers who completed the Instructor Interview Questions also completed 
instructor evaluation survey forms. The survey asked 41 questions about teachers’ use of 
PLATO, divided into three sections. Part I had 28 questions about the PLATO products 
and their use. Teachers were given a five-point scale of answers, from “Strongly Agree” 
to “Strongly Disagree.” Part II of the survey asked how often teachers gave instructions 
to students regarding the use of the PLATO system; teachers answered on a six-point 
rating scale, from 5 (daily) to 0 (never). Part III of the survey posed seven open-ended 
questions, to be answered in writing. 

PLATO Content. The first group of questions discussed here asked about the content 
of PLATO lessons and their alignment with state, district, and teacher objectives. In 
response to these questions, from one to three teachers agreed that content was good. 
Most positive responses came for content being current, course objectives aligning with 
the teacher’s own, and content being good for topics and free of errors. 
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Figure 7. PLATO Content 




Number of Teachers 



| H Strongly Agree EJAgree DNeutral □ Disagree E5 Strongly Disagree [ 



Figure 8. PLATO Content Mean Ratings 



5. Content was generally up- 
to-date. 

2. PLATO course objectives 
aligned to my own. 

4. Content seemed generally 
free of errors. 

1. PLATO content for these 
topics is good. 

8. There was adequate 
depth in exercises and tests. 

7. Students generally 
understood the explanations. 

3. PLATO content aligns 
with our standard final test. 




1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree 
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Instructional Design. Respondents were typically positive about PLATO software’s 
instructional design and interface. They were most positive about program tutorials 
involving students through interactive style and consistent use of keystrokes and display 
style. Lower ratings were given for graphics or color being used appropriately. 

Figure 9. Instructional Design 



12. PLATO used consistent keystrokes, display style. 
10. Tutorials involved students through interactive style. 

15. Screens were consistently readable. 
6. Quality, style of instruction consistent throughout. 
9. All lesson parts aligned with Instructor Guides. 
1 1 . Software was generally free of bugs or errors. 

14. Graphics were used appropriately. 
13. Color was used appropriately. 




Number of teachers 



03 Strongly Agree E3Agree □ Neutral □ Disagree B Strongly Disagree 



Figure 10. Instructional Design Mean Ratings 
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13. Color was used appropriately. 
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1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree 
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Teacher Experience with PLATO. All of the respondents said that they had been 
adequately trained to use PLATO. One of the teachers was interested in more training. All of the 
teachers agreed that they could make assignments on the system. Two teachers agreed that they 
were able to use student progress reports and to relate PLATO to classroom activities. Two of 
three teachers said that they enjoy working with PLATO, that PLATO is useful in teaching, and 
that computer work is productive. 



Figure 1 1 . Teacher Experience with PLATO 

27. I was adequately trained to use PLATO. 

18. I could make student assignments on the system. 

16. I was able to use student progress reports. 

20. I was able to relate PLATO to classroom activities. 

25. I enjoy working with the PLATO computer system. 

24. Computer work is productive, not frustrating. 

26. The PLATO system was useful in teaching. 

28. I would like more training on using PLATO well. 

0 12 3 4 

Number of Teachers 

| E3 Strongly Agree E3 Agree □ Neutral □ Disagree B Strongly Disagree] 




Figure 12. Teacher Experience Mean Ratings 
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Student Experience with PLATO. All of the teachers agreed that students had enough 
time to work on PLATO. Two of the teachers agreed that they could do tutoring while students 
used PLATO and that students are seldom confused or trapped when using PLATO. One teacher 
strongly disagreed to this last question. 

Figure 13. Student Experience with PLATO 



19. Students had enough time 
on PLATO. 



1 7. i could do tutoring while 
students used PLATO. 



23. My students respond well 
to the PLATO system. 



21 . in general, students 
respond well to PLATO. 



22. Students are seldom 
confused, trapped by PLATO. 




Number of T eachers 

E Strongly Agree E3 Agree □ Neutral □ Disagree E Strongly Disagree^ 



Figure 14. Student Experience Mean Ratings 



1 9. Students had enough time 
on PLATO. 



17. i could do tutoring while 
students used PLATO. 



23. My students respond well 
to the PLATO system. 



21. in general, students 
respond well to PLATO. 



22. Students are seldom 
confused, trapped by PLATO. 




1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree 
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Frequency Of Activities. The Instructor’s PLATO Evaluation asked for ratings of how 
frequently teachers explained six issues to their students. Teachers discussed the following issues 
during most uses of PLATO: prerequisites for success with PLATO; procedures for getting help 
on PLATO; and how PLATO fits into course goals. The other issues were discussed less often. 



Figure 15. Frequency of PLATO Activities \ 

4 
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Figure 16. Mean of PLATO Activities 
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needed to succeed with 
PLATO modules. 

34. Explain procedures for 
getting help on PLATO. 



32. Explained how PLATO fits 
into course goals. 



31. Described learning 
objectives to students. 



33. Explained incentives for 
doing well on PLATO. 

30. Helped relate PLATO 
assignments to personal 
experiences. 
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1=Never, 2=Once, 3=Beginning, 4=Occasionally, 5=Most, 6=Each 
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Conclusions & Discussion 



Conclusions will be presented and discussed in the general order that information was 
presented in the program description and data analysis section. 

For the program description, the site coordinator, teacher, and student responses (while 
incomplete) did reveal some strengths and needs in the Highland Park PLATO selection 
and implementation. 

• The Superintendent brought a new perspective about learning software when he 
joined the district, based upon, no doubt, reliable evidence. He also demonstrated 
commitment and purpose. However, it seems that administrators and teachers may 
not have been fully involved in the process of acquiring PLATO and, hence, were 
not energized about its adoption. 

• Configurations of computers in labs or classrooms, with the attendant availability 
of computer time, may pose difficulties for some teachers. District requirements 
for two hours plus per week on PLATO may not be appreciated, especially by 
teachers who have not been able to use it powerfully and therefore feel it was not 
worth their time. 

• While this sample was satisfied with their PLATO training, there is some 
indication that they could benefit from instruction in integrating PLATO and 
classroom education goals, assigning students appropriately, and making use of 
the acceleration possibilities of PLATO. Teacher turnover also necessitates 
ongoing training for technical skills as well as excellent instructional application. 

• Teachers seem to be using PLATO as supplementary to classroom instruction, 
either because of their lack of coordination with classroom objectives, student use 
problems, or software issues. 

• A good look at PLATO use from the students’ point of view could improve 
implementation. Responses hint that some students feel they are not rewarded, 
either intrinsically or extrinsically, for their PLATO experience. Teachers 
mention using more “sticks” than “carrots” with their students. 

PLATO use and test score analysis was strengthened by the large number of students 
for whom data was provided. At this time, the data exists only as a total set, without 
breakdowns for school or grade level. Below are the major findings resulting from the 
analysis. 

• PLATO was used much more in the 1998-1999 school year than in 1999- 
2000. Nevertheless, student achievement scores were relatively stable. 
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• Scale scores were higher than the state average for math, average for reading, 
and below average for writing. 

• Increases in the number of PLATO modules mastered are related to increases 
in higher test scores. 

• Increases in the number of hours spent on PLATO are related to decreases in 
test scores. This suggests that slow progress in mastering modules on PLATO 
is an indicator of learning problems that the teacher should note and address. 

• PLATO use data were modestly capable of predicting TLI scores in math, 
reading, and writing. Math regressions predicted 1 1 percent of the variance in 
TLI math scores. Reading regressions accounted for 6 percent of the 
variability in reading test scores. Writing regressions accounted for 21 percent 
of variance in the writing test scores. 



Highland Park, TX 



Page 28 



About the Authors 



David W. Quinn is currently working as an independent evaluator specializing in evaluating 
technology use for learning and teaching. He received a doctorate in educational evaluation 
from Western Michigan University in 1978. He recently completed ten years at the North 
Central Regional Educational Laboratory as a Senior Program Associate where he managed 
the evaluation unit and evaluated technology use in many setting. He has evaluated 
technology use for the states of Indiana and Virginia, and for school districts in Chicago, 
Miami-Dade, and Los Angeles County. Before NCREL, Dr. Quinn had conducted 
numerous evaluation studies for clients in K-12, university, not-for-profit social services, 
and for-profit training companies. For ten years he was on the faculty in the Department of 
Instructional Science, Brigham Young University, where he taught graduate research 
methods courses. He is the author of journal articles and book chapters evaluating 
technology use in education. 

Nancy W. Quinn is an evaluator and instructional designer. She received a master’s in 
Instructional Technology from Brigham Young University in 1991. She recently 
completed studies of statewide beginning literacy programs in three Midwestern states. 
She has evaluated the use of technology as a research tool in a research library. She has 
also evaluated the use of technology by fourth, fifth, and sixth graders in four-year at- 
school and at-home program. 



Highland Park, TX 



Page 29 



